* wip

* `rocksdb` : add byte helpers and use atomic batches (#526)

* methods use atomic write/read batches, remove flush

* update test to teardown and use async commit

* tree.get can be passed batch

* add getByteRange helper

* proof helpers all support batches

* remove stale code

* update to new storage api

* update to use peakLastTreeNode

* use deleteRange for truncations

* reopen a tree from db storage

* remove cache

* right span available nodes

* skip cache test

* skip all but merkle tree tests

* update for batch api

* rename to dbBatch for clarity

* standard

* kill stale code

* rename to read/writeBatch

* truncation should delete parent nodes

* force latest for now

* addTreeNode renamed to putTreeNode

* remove batch id from api (#530)

* rocksdb: move bitfield to db (#529)

* use rocks db storage for bitfield pages

* update tests to use rocksdb storage

* rocksdb returns buffer with length

* enable bitfield tests

* teardown storage in test helpers

* rocksdb: remove oplog and add block store API (#531)

* remove oplog and load header from rocks

* move block store to rocksdb

* remove oplog entries

* create write batches and pass to persisting methods

* comment out copyPrologue and insertBatch for now

* enable lib/core test

* write header info directly to batch

* batch updates and flush to onupdate later

* update user data api

* userData key is string in storage

* merkle tree does not own storage

* merkle tree write operations take in batch

* block operations are sync

* bitfield flush is sync and takes write batch

* pass db directly to Core.open

* set user data operates on a batch

* bitfield does not ref storage

* rocksdb: update index.js (#533)

* update index.js to use new core api

* encode manifest when storing

* tweak basic tests

* flush reader after caching

* tryFlush

---------

Co-authored-by: Mathias Buus <[email protected]>

* teardown cores implicitly in tests

* keep core.header.tree around for compat (#532)

* rocksdb: implement full API apart from batches  (#537)

* update index.js to use new core api

* encode manifest when storing

* tweak basic tests

* use opts.discoveryKey if passed

* merkle tree proof settled externally

* enable basic tests

* replicator creates batch and fulfills request separately

* ensure verify ops are using top level write batches

* bugfix: tree indexes should be multiplied by 2

* always return tree proof and await handleRequests

* update replicate tests

* always return proof request

* clear blocks now takes start and end explicitly

* update lib/multisig with new proof api

* flush manifest to batch during append

* standard

* update tests and enable as many as possible

* only finalise batch after write is flushed

* persist correct value for fork

* tidy up tests

* fix top level user data api

* update snapshot tests

* finalise reorg batch

* standard

* reorg batch is not finalised

* only clear parent nodes if ancestors > 0

* only upgrade batches need be finalised

* verifyBatchUpgrade optionally does not write

* tidy up todos

* always write manifest if we can

* fixup

* enable cache tests

* assert that merkle tree is committed before finalising

* ensure stream is drained before handling requests

* missing new keyword

* add todo for critical bugfix

* store user updates on write batch

* store pending bit sets until after flush

* use sorted list of intervals instead of map

* verify methods create their own batches

* allow updateContig to check dirty bitfield

* only one batch is allowed to be active at any time

* await flushing before processing get

* add @mafintosh interval tracker

* alloc and set using uintarray

* reenable bitfield tests

* tidy update flow and redirect for easier refactor

* add BitInterlude class

* bit interlude wraps bitfield and generates pages

* use bit interlude to calculate contig length

* optimise setting bitfield buffers

* use quickbit in bit interlude

* move all batch and flush logic into handle request

* add initial update abstraction

* extend usage of update

* move clear and clearBatch to update

* contiguous length is drop dependent

* remove last methods using write batch directly

* tidy up

* no longer need to mark pages as dirty

* remove dead props

* flushed returns if no active batch

* fixup typos

* rocksdb: copy prologue (#549)

* implement copyPrologue

* standard fixes

* review by @mafintosh

* rocksdb: add named sessions (#539)

* define scoped states per session

* return byteLength from updated batch

* pass length to createNamedSession

* add tree method for merging one tree into another

* add core method for committing state

* state stores treeLength

* session gets from state

* handle case when no tree nodes are added

* fix session get and truncate

* core truncate passes entire state down

* batch has all info after being reconciled

* update batch tests

* session uses state props

* pass whole batch to be flushed

* copy over bitfield to named session

* minor corrections

* update batch tests to be on named sessions

* update core tests

* make sure we pass state to createUpdate

* add checkout option for batches

* optionally pass discovery key to core

* each state has an independent mutex

* bitfield should full copy entire buffer

* user data is available sync for default state

* named session is always writable

* commit can specify length

* corestore owns storage

* update flushed length prop

* expose restore batch on main class

* encryption should use session state

* rebase main

* fix storage usage

* test core length is increasing

* standard fixes

* use state props and pass tree to multisig

* fix bitfield page copy

* update tests

* move to session state to dedicated abstraction

* pass parent state as capability

* ensure we have correct treeLength and up to date bitfield when opening state

* only write each bitfield page once per flush

* truncate and clear mutate treeLength

* fixes for batch tests

* enable batch tests

* overwrite if session when opts.refresh is set

* enable batch clear test

* close storage when we close the core

* storage closes automatically

* auto teardown any created core

* we have to pass an error to mutex destruction...

* close all test cores

* more missing close

* more closes

* missing session close

* close db in createIfMissing test

* more closes

* make sure all sessions are closed too

* checkout should only truncate named session

* state tracks active sessions on open and close

* screen for failing test

* core closes state

* close existing state when creating named session

* missing session close

* more closes

* pass runner to helper

* close core instead

* close state first and fix teardown order

* close state last

* missing close

* missing close

* missing close

---------

Co-authored-by: Mathias Buus <[email protected]>

* Get skips bitfield check (#559)

* wip

* adjust tests

* truncate issues a deletion of the range also

* start + 1 -> end

* rocksdb: add db snapshots (#562)

* rename clone/close to ref/unref

* add snapshot method to session state

* snapshots are not writable

* snapshotted session takes state snapshot

* fall back to core if snapshot does not have data

* review by @mafintosh

* gc snapshot

* fix snapshot teardown

* _snapshot -> storageSnapshot

* dry it a bit

---------

Co-authored-by: Mathias Buus <[email protected]>

* remove the cache, we do not use anymore (#560)

* add test for reopening writable core

* `rocksdb`: merge main (#564)


---------

Co-authored-by: HDegroote <[email protected]>
Co-authored-by: Mathias Buus <[email protected]>
Co-authored-by: rafapaezbas <[email protected]>

* always close state first

* pass opts to default storage

* Fix activity count (#568)

* add failing test

* fix the active counter with the fix from rocks branch

* do not allow core teardown during peer attachment if sessioned

* can just bump sessions instead of the extra flag

* move peer session flow fully to the makepeer lifecycle

* `rocksdb`: avoid duplicate block download (#569)

* block reqs are not readded while being processed

* check we do not have block locally before requesting

* check bitfield before deciding to wait

* add comment

---------

Co-authored-by: Mathias Buus <[email protected]>

* get user data from state storage

* tree may have been updated beneath us

* fix reorg and teardown active batches

* initial support for memory overlays (#563)

* initial support for memory overlays

* name the option dryRun

* drafts are working

* add destroy

* dont use storage helpers

* use latest storage

* internalise snapshots

* remove all memory view branches

* rocksdb: add force close option (#570)

* add force close option

* do not handle requests if core is closed

* check replicator.destroyed

* close take opts object

* _close is called with force arg instead of opts obj

* add test for force close

* force close explicitly closes all sessions

* peer checks replicator state before any db reads

* revert previous commit

* do not process any data after destroy

* upstream fix from main

* missing core update on upgrade

* use storage idle helper

* rocksdb: only create single read batch in bg call (#578)

* only create single batch per bg call

* iterate over blocks as they were read

* catch error in case uncaught

* skip read if we do not have block

* use minimum shared length (#577)

* rocksdb: commit acquires lock (#576)

* commit acquires lock directly

* check conditions after acquiring lock

* Tweak session (#579)

* remove unused args

* fix own length when core is empty

* make sure we release lock in check conflict

* copyPrologue should not call core onupdate

* tree should never be null (#584)

* rename to hypercore storage (#582)

* Add failing test (#585)

* add failing batch test

* simplify

* Rocksdb copy prologue \o/ (#586)

* new atomic single sweep copy prologue

* remove tests that use additional which is gone now

* review by @andrewosh

* helper comment

* also fwd user data to header userData

* rocksdb: check shared tree length for new sessions (#588)

* cross reference current and previous tree when opening state

* we only need to check sharedLength

* length is required in tree opening (#589)

* rename peak to peek

* Sync constructor (#591)

* wip

* checkpoint

* tests pass now

* add safe close api on core and move replicator there

* more stuff in core

* all hooks live in core

* inline replicator

* premerge

* no need to skip

* add static helper for making core cores

* bye bye non-sparse mode

* Core manages replicator lifecycle (#592)

* state unref is sync

* core closes storage

* replicator no longer needs session

* remove force close from session

* fix bad condition

* core being set is invariant

* remove stale method

* query sessions on underlying core

* move autoClose onto Core

* rename close to destroy

* core destroys state

* preload is now simply a promise

* preload promise can return opts

* no need to check if state is active

* add onidle hook

* idle waits for mutex to be free also

* remove from option

* exclusive sessions

* always emit close

* fix test

* pass userdata to create for atomicity

* core always refs default state

* session explicitly refs state

---------

Co-authored-by: Christophe Diederichs <[email protected]>
Co-authored-by: Christophe Diederichs <[email protected]>

* rocksdb: move `MemoryOverlay` to storage module (#580)

* add memory overlay test

* fix put

* implement memory overlay block deletion

* add memory overlay block deletion tests

* move tip list to abstraction and add tests

* treeNodes use tip list

* move memory overlay to hypercore-on-the-rocks

* rename peak to peek

* rocksdb: commit overwrites full state (#587)

* commit acquires lock directly

* check conditions after acquiring lock

* update dependency in place and delete blocks

* update dependency on truncate and clear if necessary

* move memory overlay to hypercore-on-the-rocks

* rename peak to peek

* use dependencies to compute flushed length

* bit-interlude supports multiple deletions

* add explicit state overwrite method

* memory overlay always has flushedLength as -1

* signature is verified against committed tree

* use flat-tree patch method

* review from @mafintosh

* only one call to splice

* rocksdb: update bitfield setRange API  (#593)

* change bitfield setRange api to start + end

* update usage of remote bitfield

* bitfield doesnt peek anymore (#595)

* fix bitfield page fill typo

* Fix events on rocks (#596)

* easy ones

* do not rely on events to adopt encryption compat

* snapshot compat length lives on the session now

* wip

* last stretch

* remove dead code

* missed one

* rocksdb: decouple session state from bitfield (#601)

* BitInterlude does not need underlying bitfield

* factor out bitfield from session state

* update tests

* update audit to use batch directly

* optimise hot paths in has

* updateContigBatch returns -1 as default

* review by @mafintosh

* rocksdb: support memory views (#602)

* move createSession onto state

* need to special case draft state

* do not throw error if we are closing

* some missing idle checks

* add weak session support (#603)

* fix dbl idle on destroy and weak session list

* Expose user data stream from lib/core.js (#604)

* expose user data stream from lib/core.js

* enclose ternary condition

* add user data stream tests

* rename to createUserDataStream

* Add snapshotFrom method (#607)

* add tree snapshot

* snapshots store explicit tree snapshot

* add snapshotFrom method

* session update does not affect snapshot state

* refactor snapshot update test

* allow root session to define session list (#608)

* allow root session to define session list

* review by @chm-diederichs

* add ongc hook

* also set gc handler when making sessions

* rocksdb: use storage API for updating dependency (#610)

* add test for snapshot on named batch

* move dependency mutation to storage api

* inherit preload promise in sessions (#606)

* inherit more from parent (#612)

* inherit more from parent

* simplify

* rocksdb: minor tweaks (#613)

* drafts are writable

* session overwrite should emit events

* rocksdb: SessionState.moveTo API (#609)

* add initial moveTo API

* snapshot makes full tree clone

* add test for snapshots

* parent state may be swapped during ready

* session has method for transferring

* only check compat if block is not local

* add snapshot move to checks

* retrying requests on session move

* transfer all replicating io when moving sessions

* drafts are writable

* session overwrite should emit events

* add option for overwrite to update dependency

* do not create snapshot over existing snapshot

* ref session state in constructor

* fix non writable named sessions

* emit migrate event on moveTo

---------

Co-authored-by: Mathias Buus <[email protected]>

* rename flushedLength to signedLength (#614)

* rocksdb: atomic operations (#615)

* optionally pass atomizer to append

* commit and overwrite accept atomizer

* commit errors on fail and destroys atomizer

* flushWriteBatch always uses active batch

* add test for overwrite

* setUserData accepts atomizer

* throw error after destroying atomizer

* standard test

* refactor mutex to return lock

* atomizer acquires lock

* add tests with concurrent operations

* fix test condition

* rewrite tests to be on same storage

* enable atomic tests

* rename atomizer to atom

* moveTo supports atomicity

* moveTo acquires lock

* fix test

* fix multiple addMonitor calls and extension registration

* replicator only set post open

* try to fix most of the moveTo state issues

* throw error if destroyed in overwrite

* bump fork on batch truncation

* do not check fork on commit as we already check tree compat and commit to main fork

* preserve fork in moveTo

* set core encryption also on setEncryptionKey

* encryption is global atm, skip test that says otherwise

* explicit corestorage check

* no need to dbl check

* rocksdb: optional argument for overwrite to infill (#617)

* add optional overwrite argument for copying all data

* missing argument

* rename option to shallow

* default to not shallow

* rocksdb: move merkle state onto session (#618)

* move merkle tree state onto the session

* add setRoots helper

* create merkle tree batches independently

* seek still needs session

* remove dead code

* flake fixed

* rocksdb: atomic sessions (#619)

* integrate new storage api

* remove old atom code

* make session operations support atomicity

* remove memory-overlay

* storage renmae batch to session

* test multiple atomic appends

* update iterator arg

* atomicity fixes

* make commit atomic

* store atom on session

* support atomic moveTo operations

* use async resume session in atomic flush

* rewrite merkle tree tests

* tidy up merkle tree

* update batch tests to use session.commit api

* move draft tests into atomic tests

* tidy up tests

* unskip batch tests

* fix copyPrologue update sweep

* revert mutex change

* rename to storage batches to rx/tx

* remove lib/oplog

* bump to storage v1

* only update state after flush is called

* session commit api takes in a session

* atom is stored on storage

* atomize session in move to if needed

* fix and enable move-to tests

* ensure atomic cores are closed

* add test for multiple flushes

---------

Co-authored-by: Christophe Diederichs <[email protected]>
Co-authored-by: Christophe Diederichs <[email protected]>
Co-authored-by: HDegroote <[email protected]>
Co-authored-by: rafapaezbas <[email protected]>